-
Open
-
Open
-
Open
-
Open
-
Open
-
Open
Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources: A Hierarchical Agent-based Approach. (arXiv:2309.06604v2 [cs.LG] UPDATED)
Algorithm selection and hyperparameter tuning are critical steps in both
academic and applied machine learning. On the other hand, these steps are
becoming ever increasingly delicate due to the extensive rise in the number,
diversity, and distributedness of machine learning resources. Multi-agent
systems, when applied to the design of machine learning platforms, bring about
several distinctive characteristics such as scalability, flexibility, and
robustness, just to name a few. This paper proposes a fully automatic and
collaborative agent-based mechanism for selecting distributedly organized
machine learning algorithms and simultaneously tuning their hyperparameters.
Our method builds upon an existing agent-based hierarchical machine-learning
platform and augments its query structure to support the aforementioned
functionalities without being limited to specific learning, selection, and
tuning mechanisms. We have conducted theoretical assessments, formal
verification, and analytical study to demonstrate the correctness, resource
utilization, and computational efficiency of our technique. According to the
results, our solution is totally correct and exhibits linear time and space
complexity in relation to the size of available resources. To provide concrete
examples of how the proposed methodologies can effectively adapt and perform
across a range of algorithmic options and datasets, we have also conducted a
series of experiments using a system comprised of 24 algorithms and 9 datasets.
( 3
min )
Nowhere coexpanding functions. (arXiv:2303.12814v3 [stat.ML] UPDATED)
We define a family of $C^1$ functions which we call "nowhere coexpanding
functions" that is closed under composition and includes all $C^3$ functions
with non-positive Schwarzian derivative. We establish results on the number and
nature of the fixed points of these functions, including a generalisation of a
classic result of Singer.
( 2
min )
Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing. (arXiv:2309.04612v2 [cs.LG] UPDATED)
Feature generation aims to generate new and meaningful features to create a
discriminative representation space.A generated feature is meaningful when the
generated feature is from a feature pair with inherent feature interaction. In
the real world, experienced data scientists can identify potentially useful
feature-feature interactions, and generate meaningful dimensions from an
exponentially large search space, in an optimal crossing form over an optimal
generation path. But, machines have limited human-like abilities.We generalize
such learning tasks as self-optimizing feature generation. Self-optimizing
feature generation imposes several under-addressed challenges on existing
systems: meaningful, robust, and efficient generation. To tackle these
challenges, we propose a principled and generic representation-crossing
framework to solve self-optimizing feature generation.To achieve hashing
representation, we propose a three-step approach: feature discretization,
feature hashing, and descriptive summarization. To achieve reinforcement
crossing, we develop a hierarchical reinforcement feature crossing approach.We
present extensive experimental results to demonstrate the effectiveness and
efficiency of the proposed method. The code is available at
https://github.com/yingwangyang/HRC_feature_cross.git.
( 2
min )
Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary tasks. (arXiv:2309.07794v1 [cs.CL])
Effectively leveraging multimodal information from social media posts is
essential to various downstream tasks such as sentiment analysis, sarcasm
detection and hate speech classification. However, combining text and image
information is challenging because of the idiosyncratic cross-modal semantics
with hidden or complementary information present in matching image-text pairs.
In this work, we aim to directly model this by proposing the use of two
auxiliary losses jointly with the main task when fine-tuning any pre-trained
multimodal model. Image-Text Contrastive (ITC) brings image-text
representations of a post closer together and separates them from different
posts, capturing underlying dependencies. Image-Text Matching (ITM) facilitates
the understanding of semantic correspondence between images and text by
penalizing unrelated pairs. We combine these objectives with five multimodal
models, demonstrating consistent improvements across four popular social media
datasets. Furthermore, through detailed analysis, we shed light on the specific
scenarios and cases where each auxiliary task proves to be most effective.
( 2
min )
Reasoning with Language Model Prompting: A Survey. (arXiv:2212.09597v7 [cs.CL] UPDATED)
Reasoning, as an essential ability for complex problem-solving, can provide
back-end support for various real-world applications, such as medical
diagnosis, negotiation, etc. This paper provides a comprehensive survey of
cutting-edge research on reasoning with language model prompting. We introduce
research works with comparisons and summaries and provide systematic resources
to help beginners. We also discuss the potential reasons for emerging such
reasoning abilities and highlight future research directions. Resources are
available at https://github.com/zjunlp/Prompt4ReasoningPapers (updated
periodically).
( 2
min )
Understanding Vector-Valued Neural Networks and Their Relationship with Real and Hypercomplex-Valued Neural Networks. (arXiv:2309.07716v1 [cs.LG])
Despite the many successful applications of deep learning models for
multidimensional signal and image processing, most traditional neural networks
process data represented by (multidimensional) arrays of real numbers. The
intercorrelation between feature channels is usually expected to be learned
from the training data, requiring numerous parameters and careful training. In
contrast, vector-valued neural networks are conceived to process arrays of
vectors and naturally consider the intercorrelation between feature channels.
Consequently, they usually have fewer parameters and often undergo more robust
training than traditional neural networks. This paper aims to present a broad
framework for vector-valued neural networks, referred to as V-nets. In this
context, hypercomplex-valued neural networks are regarded as vector-valued
models with additional algebraic properties. Furthermore, this paper explains
the relationship between vector-valued and traditional neural networks.
Precisely, a vector-valued neural network can be obtained by placing
restrictions on a real-valued model to consider the intercorrelation between
feature channels. Finally, we show how V-nets, including hypercomplex-valued
neural networks, can be implemented in current deep-learning libraries as
real-valued networks.
( 2
min )
Random Feature Amplification: Feature Learning and Generalization in Neural Networks. (arXiv:2202.07626v4 [cs.LG] UPDATED)
In this work, we provide a characterization of the feature-learning process
in two-layer ReLU networks trained by gradient descent on the logistic loss
following random initialization. We consider data with binary labels that are
generated by an XOR-like function of the input features. We permit a constant
fraction of the training labels to be corrupted by an adversary. We show that,
although linear classifiers are no better than random guessing for the
distribution we consider, two-layer ReLU networks trained by gradient descent
achieve generalization error close to the label noise rate. We develop a novel
proof technique that shows that at initialization, the vast majority of neurons
function as random features that are only weakly correlated with useful
features, and the gradient descent dynamics 'amplify' these weak, random
features to strong, useful features.
( 2
min )
deepFDEnet: A Novel Neural Network Architecture for Solving Fractional Differential Equations. (arXiv:2309.07684v1 [cs.LG])
The primary goal of this research is to propose a novel architecture for a
deep neural network that can solve fractional differential equations
accurately. A Gaussian integration rule and a $L_1$ discretization technique
are used in the proposed design. In each equation, a deep neural network is
used to approximate the unknown function. Three forms of fractional
differential equations have been examined to highlight the method's
versatility: a fractional ordinary differential equation, a fractional order
integrodifferential equation, and a fractional order partial differential
equation. The results show that the proposed architecture solves different
forms of fractional differential equations with excellent precision.
( 2
min )
A Novel Local-Global Feature Fusion Framework for Body-weight Exercise Recognition with Pressure Mapping Sensors. (arXiv:2309.07888v1 [cs.CV])
We present a novel local-global feature fusion framework for body-weight
exercise recognition with floor-based dynamic pressure maps. One step further
from the existing studies using deep neural networks mainly focusing on global
feature extraction, the proposed framework aims to combine local and global
features using image processing techniques and the YOLO object detection to
localize pressure profiles from different body parts and consider physical
constraints. The proposed local feature extraction method generates two sets of
high-level local features consisting of cropped pressure mapping and numerical
features such as angular orientation, location on the mat, and pressure area.
In addition, we adopt a knowledge distillation for regularization to preserve
the knowledge of the global feature extraction and improve the performance of
the exercise recognition. Our experimental results demonstrate a notable 11
percent improvement in F1 score for exercise recognition while preserving
label-specific features.
( 2
min )
Survival Estimation for Missing not at Random Censoring Indicators based on Copula Models. (arXiv:2009.01726v2 [stat.ML] UPDATED)
In the presence of right-censored data with covariates, the conditional
Kaplan-Meier estimator (also known as the Beran estimator) consistently
estimates the conditional survival function of the random follow-up for the
event of interest. However, a necessary condition is the unambiguous knowledge
of whether each individual is censored or not, which may be incomplete in
practice. We therefore propose a study of the Beran estimator when the
censoring indicators are generic random variables and discuss necessary
conditions for the efficiency of the Beran estimator. From this, we provide a
new estimator for the conditional survival function with missing not at random
(MNAR) censoring indicators based on a conditional copula model for the
missingness mechanism. In addition to the theoretical results, we illustrate
how the estimators work for small samples through a simulation study and show
their practical applicability by analyzing synthetic and real data.
( 2
min )
Communication Efficient Private Federated Learning Using Dithering. (arXiv:2309.07809v1 [cs.LG])
The task of preserving privacy while ensuring efficient communication is a
fundamental challenge in federated learning. In this work, we tackle this
challenge in the trusted aggregator model, and propose a solution that achieves
both objectives simultaneously. We show that employing a quantization scheme
based on subtractive dithering at the clients can effectively replicate the
normal noise addition process at the aggregator. This implies that we can
guarantee the same level of differential privacy against other clients while
substantially reducing the amount of communication required, as opposed to
transmitting full precision gradients and using central noise addition. We also
experimentally demonstrate that the accuracy of our proposed approach matches
that of the full precision gradient method.
( 2
min )
On a continuous time model of gradient descent dynamics and instability in deep learning. (arXiv:2302.01952v3 [stat.ML] UPDATED)
The recipe behind the success of deep learning has been the combination of
neural networks and gradient-based optimization. Understanding the behavior of
gradient descent however, and particularly its instability, has lagged behind
its empirical success. To add to the theoretical tools available to study
gradient descent we propose the principal flow (PF), a continuous time flow
that approximates gradient descent dynamics. To our knowledge, the PF is the
only continuous flow that captures the divergent and oscillatory behaviors of
gradient descent, including escaping local minima and saddle points. Through
its dependence on the eigendecomposition of the Hessian the PF sheds light on
the recently observed edge of stability phenomena in deep learning. Using our
new understanding of instability we propose a learning rate adaptation method
which enables us to control the trade-off between training stability and test
set evaluation performance.
( 2
min )
Latent Representation and Simulation of Markov Processes via Time-Lagged Information Bottleneck. (arXiv:2309.07200v1 [cs.LG])
Markov processes are widely used mathematical models for describing dynamic
systems in various fields. However, accurately simulating large-scale systems
at long time scales is computationally expensive due to the short time steps
required for accurate integration. In this paper, we introduce an inference
process that maps complex systems into a simplified representational space and
models large jumps in time. To achieve this, we propose Time-lagged Information
Bottleneck (T-IB), a principled objective rooted in information theory, which
aims to capture relevant temporal features while discarding high-frequency
information to simplify the simulation task and minimize the inference error.
Our experiments demonstrate that T-IB learns information-optimal
representations for accurately modeling the statistical properties and dynamics
of the original process at a selected time lag, outperforming existing
time-lagged dimensionality reduction methods.
( 2
min )
Learning Quasi-Static 3D Models of Markerless Deformable Linear Objects for Bimanual Robotic Manipulation. (arXiv:2309.07609v1 [cs.RO])
The robotic manipulation of Deformable Linear Objects (DLOs) is a vital and
challenging task that is important in many practical applications. Classical
model-based approaches to this problem require an accurate model to capture how
robot motions affect the deformation of the DLO. Nowadays, data-driven models
offer the best tradeoff between quality and computation time. This paper
analyzes several learning-based 3D models of the DLO and proposes a new one
based on the Transformer architecture that achieves superior accuracy, even on
the DLOs of different lengths, thanks to the proposed scaling method. Moreover,
we introduce a data augmentation technique, which improves the prediction
performance of almost all considered DLO data-driven models. Thanks to this
technique, even a simple Multilayer Perceptron (MLP) achieves close to
state-of-the-art performance while being significantly faster to evaluate. In
the experiments, we compare the performance of the learning-based 3D models of
the DLO on several challenging datasets quantitatively and demonstrate their
applicability in the task of shaping a DLO.
( 2
min )
Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speech. (arXiv:2110.12539v3 [cs.SD] UPDATED)
We present a Split Vector Quantized Variational Autoencoder (SVQ-VAE)
architecture using a split vector quantizer for NTTS, as an enhancement to the
well-known Variational Autoencoder (VAE) and Vector Quantized Variational
Autoencoder (VQ-VAE) architectures. Compared to these previous architectures,
our proposed model retains the benefits of using an utterance-level bottleneck,
while keeping significant representation power and a discretized latent space
small enough for efficient prediction from text. We train the model on
recordings in the expressive task-oriented dialogues domain and show that
SVQ-VAE achieves a statistically significant improvement in naturalness over
the VAE and VQ-VAE models. Furthermore, we demonstrate that the SVQ-VAE latent
acoustic space is predictable from text, reducing the gap between the standard
constant vector synthesis and vocoded recordings by 32%.
( 2
min )
MAHTM: A Multi-Agent Framework for Hierarchical Transactive Microgrids. (arXiv:2303.08447v2 [cs.LG] UPDATED)
Integrating variable renewable energy into the grid has posed challenges to
system operators in achieving optimal trade-offs among energy availability,
cost affordability, and pollution controllability. This paper proposes a
multi-agent reinforcement learning framework for managing energy transactions
in microgrids. The framework addresses the challenges above: it seeks to
optimize the usage of available resources by minimizing the carbon footprint
while benefiting all stakeholders. The proposed architecture consists of three
layers of agents, each pursuing different objectives. The first layer,
comprised of prosumers and consumers, minimizes the total energy cost. The
other two layers control the energy price to decrease the carbon impact while
balancing the consumption and production of both renewable and conventional
energy. This framework also takes into account fluctuations in energy demand
and supply.
( 2
min )
Some notes concerning a generalized KMM-type optimization method for density ratio estimation. (arXiv:2309.07887v1 [cs.LG])
In the present paper we introduce new optimization algorithms for the task of
density ratio estimation. More precisely, we consider extending the well-known
KMM method using the construction of a suitable loss function, in order to
encompass more general situations involving the estimation of density ratio
with respect to subsets of the training data and test data, respectively. The
associated codes can be found at https://github.com/CDAlecsa/Generalized-KMM.
( 2
min )
Correcting sampling biases via importance reweighting for spatial modeling. (arXiv:2309.04824v2 [cs.LG] UPDATED)
In machine learning models, the estimation of errors is often complex due to
distribution bias, particularly in spatial data such as those found in
environmental studies. We introduce an approach based on the ideas of
importance sampling to obtain an unbiased estimate of the target error. By
taking into account difference between desirable error and available data, our
method reweights errors at each sample point and neutralizes the shift.
Importance sampling technique and kernel density estimation were used for
reweighteing. We validate the effectiveness of our approach using artificial
data that resemble real-world spatial datasets. Our findings demonstrate
advantages of the proposed approach for the estimation of the target error,
offering a solution to a distribution shift problem. Overall error of
predictions dropped from 7% to just 2% and it gets smaller for larger samples.
( 2
min )
HurriCast: An Automatic Framework Using Machine Learning and Statistical Modeling for Hurricane Forecasting. (arXiv:2309.07174v1 [cs.LG])
Hurricanes present major challenges in the U.S. due to their devastating
impacts. Mitigating these risks is important, and the insurance industry is
central in this effort, using intricate statistical models for risk assessment.
However, these models often neglect key temporal and spatial hurricane patterns
and are limited by data scarcity. This study introduces a refined approach
combining the ARIMA model and K-MEANS to better capture hurricane trends, and
an Autoencoder for enhanced hurricane simulations. Our experiments show that
this hybrid methodology effectively simulate historical hurricane behaviors
while providing detailed projections of potential future trajectories and
intensities. Moreover, by leveraging a comprehensive yet selective dataset, our
simulations enrich the current understanding of hurricane patterns and offer
actionable insights for risk management strategies.
( 2
min )
LambdaKG: A Library for Pre-trained Language Model-Based Knowledge Graph Embeddings. (arXiv:2210.00305v3 [cs.CL] UPDATED)
Knowledge Graphs (KGs) often have two characteristics: heterogeneous graph
structure and text-rich entity/relation information. Text-based KG embeddings
can represent entities by encoding descriptions with pre-trained language
models, but no open-sourced library is specifically designed for KGs with PLMs
at present. In this paper, we present LambdaKG, a library for KGE that equips
with many pre-trained language models (e.g., BERT, BART, T5, GPT-3), and
supports various tasks (e.g., knowledge graph completion, question answering,
recommendation, and knowledge probing). LambdaKG is publicly open-sourced at
https://github.com/zjunlp/PromptKG/tree/main/lambdaKG, with a demo video at
this http URL and long-term maintenance.
( 2
min )
Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis. (arXiv:2309.07168v1 [cs.LG])
Open-ended learning benefits immensely from the use of symbolic methods for
goal representation as they offer ways to structure knowledge for efficient and
transferable learning. However, the existing Hierarchical Reinforcement
Learning (HRL) approaches relying on symbolic reasoning are often limited as
they require a manual goal representation. The challenge in autonomously
discovering a symbolic goal representation is that it must preserve critical
information, such as the environment dynamics. In this work, we propose a
developmental mechanism for subgoal discovery via an emergent representation
that abstracts (i.e., groups together) sets of environment states that have
similar roles in the task. We create a HRL algorithm that gradually learns this
representation along with the policies and evaluate it on navigation tasks to
show the learned representation is interpretable and results in data
efficiency.
( 2
min )
-
Open
Joint Community Detection and Rotational Synchronization via Semidefinite Programming. (arXiv:2105.06031v2 [stat.ML] UPDATED)
In the presence of heterogeneous data, where randomly rotated objects fall
into multiple underlying categories, it is challenging to simultaneously
classify them into clusters and synchronize them based on pairwise relations.
This gives rise to the joint problem of community detection and
synchronization. We propose a series of semidefinite relaxations, and prove
their exact recovery when extending the celebrated stochastic block model to
this new setting where both rotations and cluster identities are to be
determined. Numerical experiments demonstrate the efficacy of our proposed
algorithms and confirm our theoretical result which indicates a sharp phase
transition for exact recovery.
( 2
min )
Deep Nonparametric Convexified Filtering for Computational Photography, Image Synthesis and Adversarial Defense. (arXiv:2309.06724v2 [cs.CV] UPDATED)
We aim to provide a general framework of for computational photography that
recovers the real scene from imperfect images, via the Deep Nonparametric
Convexified Filtering (DNCF). It is consists of a nonparametric deep network to
resemble the physical equations behind the image formation, such as denoising,
super-resolution, inpainting, and flash. DNCF has no parameterization dependent
on training data, therefore has a strong generalization and robustness to
adversarial image manipulation. During inference, we also encourage the network
parameters to be nonnegative and create a bi-convex function on the input and
parameters, and this adapts to second-order optimization algorithms with
insufficient running time, having 10X acceleration over Deep Image Prior. With
these tools, we empirically verify its capability to defend image
classification deep networks against adversary attack algorithms in real-time.
( 2
min )
Survival Estimation for Missing not at Random Censoring Indicators based on Copula Models. (arXiv:2009.01726v2 [stat.ML] UPDATED)
In the presence of right-censored data with covariates, the conditional
Kaplan-Meier estimator (also known as the Beran estimator) consistently
estimates the conditional survival function of the random follow-up for the
event of interest. However, a necessary condition is the unambiguous knowledge
of whether each individual is censored or not, which may be incomplete in
practice. We therefore propose a study of the Beran estimator when the
censoring indicators are generic random variables and discuss necessary
conditions for the efficiency of the Beran estimator. From this, we provide a
new estimator for the conditional survival function with missing not at random
(MNAR) censoring indicators based on a conditional copula model for the
missingness mechanism. In addition to the theoretical results, we illustrate
how the estimators work for small samples through a simulation study and show
their practical applicability by analyzing synthetic and real data.
( 2
min )
Convergence analysis of online algorithms for vector-valued kernel regression. (arXiv:2309.07779v1 [stat.ML])
We consider the problem of approximating the regression function from noisy
vector-valued data by an online learning algorithm using an appropriate
reproducing kernel Hilbert space (RKHS) as prior. In an online algorithm,
i.i.d. samples become available one by one by a random process and are
successively processed to build approximations to the regression function. We
are interested in the asymptotic performance of such online approximation
algorithms and show that the expected squared error in the RKHS norm can be
bounded by $C^2 (m+1)^{-s/(2+s)}$, where $m$ is the current number of processed
data, the parameter $0<s\leq 1$ expresses an additional smoothness assumption
on the regression function and the constant $C$ depends on the variance of the
input noise, the smoothness of the regression function and further parameters
of the algorithm.
( 2
min )
Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data. (arXiv:2202.05928v4 [cs.LG] UPDATED)
Benign overfitting, the phenomenon where interpolating models generalize well
in the presence of noisy data, was first observed in neural network models
trained with gradient descent. To better understand this empirical observation,
we consider the generalization error of two-layer neural networks trained to
interpolation by gradient descent on the logistic loss following random
initialization. We assume the data comes from well-separated class-conditional
log-concave distributions and allow for a constant fraction of the training
labels to be corrupted by an adversary. We show that in this setting, neural
networks exhibit benign overfitting: they can be driven to zero training error,
perfectly fitting any noisy training labels, and simultaneously achieve minimax
optimal test error. In contrast to previous work on benign overfitting that
require linear or kernel-based predictors, our analysis holds in a setting
where both the model and learning dynamics are fundamentally nonlinear.
( 2
min )
On a continuous time model of gradient descent dynamics and instability in deep learning. (arXiv:2302.01952v3 [stat.ML] UPDATED)
The recipe behind the success of deep learning has been the combination of
neural networks and gradient-based optimization. Understanding the behavior of
gradient descent however, and particularly its instability, has lagged behind
its empirical success. To add to the theoretical tools available to study
gradient descent we propose the principal flow (PF), a continuous time flow
that approximates gradient descent dynamics. To our knowledge, the PF is the
only continuous flow that captures the divergent and oscillatory behaviors of
gradient descent, including escaping local minima and saddle points. Through
its dependence on the eigendecomposition of the Hessian the PF sheds light on
the recently observed edge of stability phenomena in deep learning. Using our
new understanding of instability we propose a learning rate adaptation method
which enables us to control the trade-off between training stability and test
set evaluation performance.
( 2
min )
